DVC principles

Large files (datasets, models…) are kept outside Git
Each large file or directory put under DVC tracking has an associated .dvc file
Git only tracks the .dvc files (metadata)

Installation

• pip:

  pip install dvc

• conda

• pipx (if you want dvc available everywhere without having to activate virtual envs):

  pipx install dvc

Optional dependencies [s3], [gdrive], etc. for remote storage

How to run

• Terminal

  dvc ...

• VS Code extension

• Python library if installed via pip or conda

  import dvc.api

Acknowledgements

Code and data for this webinar modified from:

• Real Python
• DataLad handbook
• DVC documentation

The project

tree -L 3

├── LICENSE
├── data
│   ├── prepared
│   └── raw
│       ├── train
│       └── val
├── metrics
├── model
├── requirements.txt
└── src
    ├── evaluate.py
    ├── prepare.py
    └── train.py

Initialize Git repo

git init

Initialized empty Git repository in dvc/.git/

git status

On branch main

No commits yet

Untracked files:
    LICENSE
    data/
    requirements.txt
    src/

Initialize DVC project

dvc init

Initialized DVC repository.

You can now commit the changes to git.

Commit DVC system files

git status

On branch main

No commits yet

Changes to be committed:
    new file:   .dvc/.gitignore
    new file:   .dvc/config
    new file:   .dvcignore

Untracked files:
    LICENSE
    data/
    requirements.txt
    src/

So we can directly commit:

git commit -m "Initialize DVC"

Prepare repo

Let’s work in a virtual environment:

python -m venv venv && echo venv > .gitignore
source venv/bin/activate

Let’s clean things up:

git add .gitignore LICENSE requirements.txt
git commit -m "Add general files"
git add src
git commit -m "Add scripts"

git status

On branch main
Untracked files:
    data/

Put data under DVC tracking

We are still not tracking any data:

dvc status

There are no data or pipelines tracked in this project yet.

You can choose what to track as a unit (i.e. each picture individually, the whole data directory as a unit)

Let’s break it down by set:

dvc add data/raw/train
dvc add data/raw/val

This added data to .dvc/cache and created 3 files in data/raw:

• .gitignore
• train.dvc
• val.dvc

The .gitignore tells Git not to track the data:

cat data/raw/.gitignore

/train
/val

The .dvc files contain the metadata for the cached directories

Tracked data

We are all good:

dvc status

Data and pipelines are up to date.

Data (de)duplication

Link between checked-out version of a file/directory and the cache:

*Reflinks only available for a few file systems (Btrfs, XFS, OCFS2, or APFS)

Commit the metafiles

The metafiles should be put under Git version control

dvc config [--system] [--global] core.autostage true

You can then commit directly:

git commit -m "Initial version of data"
git status

On branch main
nothing to commit, working tree clean

Track changes to the data

Let’s make some change to the data:

rm data/raw/val/n03445777/ILSVRC2012_val*

git status

On branch main
nothing to commit, working tree clean

dvc status

data/raw/val.dvc:
    changed outs:
            modified:           data/raw/val

Add changes to DVC

dvc add data/raw/val
dvc status

Data and pipelines are up to date.

git status

On branch main
Changes to be committed:
    modified:   data/raw/val.dvc

git commit -m "Delete data/raw/val/n03445777/ILSVRC2012_val*"

Check out older versions

What if we want to go back to the 1st version of our data?

For this, we first use Git to checkout the proper commit, then run dvc checkout to have the data catch up to the .dvc file

To avoid forgetting to run the commands that will make DVC catch up to Git, we can automate this process by installing Git hooks:

dvc install

Git workflows

git checkout is ok to have a look, but a detached HEAD is not a good place to create new commits

Let’s create a new branch and switch to it:

git switch -c alternative

Switched to a new branch 'alternative'

Classic workflow

The Git project (including .dvc files) go to a Git remote (GitHub/GitLab/Bitbucket/server)

The data go to a DVC remote (AWS/Azure/Google Drive/server/etc.)

DVC remotes

DVC can use many cloud storage or remote machines/server via SSH, WebDAV, etc.

Let’s create a local remote here:

# Create a directory outside the project
mkdir ../remote

# Setup default (-d) remote
dvc remote add -d local_remote ../remote

Setting 'local_remote' as a default remote.

cat .dvc/config

[core]
    remote = local_remote
['remote "local_remote"']
    url = ../../remote

Commit remote config

The new remote configuration should be committed:

git status

On branch alternative

Changes not staged for commit:
    modified:   .dvc/config

git add .
git commit -m "Config remote"

Push to remotes

Let’s push the data from the cache (.dvc/cache) to the remote:

dvc push

2702 files pushed

By default, the entire data cache gets pushed to the remote, but there are many options

dvc push data/raw/val.dvc

Pull from remotes

dvc fetch downloads data from the remote into the cache. To have it update the working directory, follow by dvc checkout

You can do these 2 commands at the same time with dvc pull

DVC pipelines

DVC pipelines create reproducible workflows and are functionally similar to Makefiles

Each step in a pipeline is created with dvc stage add and add an entry to a dvc.yaml file

This entry contains:

• cmd: the command executed
• deps: the dependencies
• outs: the outputs

The file is then used to visualize the pipeline and run it

Example

Let’s create a pipeline to run a classifier on our data

The pipeline contains 3 steps:

• prepare
• train
• evaluate

1^st stage (data preparation)

dvc stage add -n prepare \
    -d src/prepare.py -d data/raw \
    -o data/prepared/train.csv -o data/prepared/test.csv \
    python src/prepare.py

Added stage 'prepare' in 'dvc.yaml'

2^nd stage (training)

dvc stage add -n train \
    -d src/train.py -d data/prepared/train.csv \
    -o model/model.joblib \
    python src/train.py

Added stage `train` in 'dvc.yaml'

3^rd stage (evaluation)

dvc stage add -n evaluate \
    -d src/evaluate.py -d model/model.joblib \
    -M metrics/accuracy.json \
    python src/evaluate.py

Added stage `evaluate` in 'dvc.yaml'

Commit pipeline

git commit -m "Define pipeline"

Visualize pipeline

dvc dag

 +---------+
 | prepare |
 +---------+
      *
      *
      *
  +-------+
  | train |
  +-------+
      *
      *
      *
 +----------+
 | evaluate |
 +----------+

Run pipeline

dvc repro

git commit -m "First pipeline run"

Going further … next time

DVC is a sophisticated tool with many additional features:

• Creation of data registries

• DVCLive

  A Python library to log experiment metrics

• Visualize the performance logs as plots

• Continuous integration

  With the sister project CML (Continuous Machine Learning)